Quality of data (how to ensure the impressions correspond to reality and are backed by several people)?
Rajasthan
Not every progress marker needs to be developed for every boundary partner. 1. Role and importance of OM or what did you do instead and why?
Existing information in: (--> Address the info progress markers)
FG monthly reports
Group meeting registers
Weight records (kids and bucks)
Buck service records
Baseline survey info
IP reports
All this is at producer level & FG (service provider): not yet reached the level of where we could monitor post production and enabling agencies
Importance of information:
Helps in monitoring progress of field activities
Since this is monthly information it gives flexiblity to make changes after review meeting every month.
Monitoring FG performance
2. What modifications brought to the OM methodology?
We made changes to group meeting registers for easy compilation. Right now it is a long narrative done in local language. Now we look at key issues, what issues to take up in IP meetings, what issues to bring from IP to group meetings, formatted in bullet points (easier to compile later).
3. Plan to gather monitoring data?
Collection of data at Field Guide (FG) level:
100 households/FG;
8-10 group meeting reports/FG.
Total data: 2700 HH / month.
Data entry at Jhadol (field): (we now have a field officer entering this data).
SN Excel, listing issues highlighted
Coding
Data compilation at village level - at household level? The local field officer now has that information but entering information at HH level is very time-consuming.
Analysing monthly:
The different activities
Progress
Feedback to FGs
E.g. we're looking at whether our group is meeting without any facilitator. It's still too early to say if this works but now, after running tests at village level, we find out that some issues were addressed in villages (or not) so we get an idea of whether a group is active and mature or not.
Now we also get the amount of goats per sale but we can only say once a year if someone is selling more goats (otherwise it would be too time consuming).
The timeline is an important issue to keep in consideration.
Jharkhand has a similar approach.
Q: Are there any plans to move beyond production to trace changes among other boundary partners? A: We haven't reached the state where traders come and purchase. We will have traders come and buy at the end of this year only. We are collecting market information etc. and we will provide this info to traders later. We will capture other information later. Q: To what extent do existing forms capture other progress markers? Does this allow you to look at a set of progress markers? A: If we have achieved 50%, we think it's a high level change. The information is going to group members but not many issues are flagged for consideration in IPs. That part is weak. (who is checking if progress markers are achieved? A: I (Ramkumar) am entering the data and provide guidance to code and compile this into the framework). Q: Monthly analysis is a lot of work - we should discuss this... Is it really useful? Is it worth it? A: This info is already collected by BAIF and we continue to use it. We have good discussions with FGs now because we can point out to issues we see from that data. This goes beyond anecdotal information. Field guides are key. Q: Can you say more about the progress markers identified? A: When people follow action plans etc. it's another progress marker. There is a weakness we identified about this marker. Information flows from IP to the groups happen but not the other way around, so we assess ourselves poorly about that. Q: Are there groups that have reached the state of 'adult groups'? A: That will happen. Group maturity is taking longer than anticipated. Q: You are collecting a lot of quantitative data but shouldn't you also measure the internal management capacity of the group? A: We have some action plans prepared in IP meetings. You have some information about why some people participated to meetings or not etc. We are now coding some of these statements. Q: Who is collecting the data? A: Field guides collect all data Q: You gave them training? A: Yes
IP reports: field staff prepare this in Hindi and send it to Ramkumar for
What we did: Before using the journal we had a team meeting with the objective 'to discover what OM information the team can come up with and test if the idea of a monthly meeting to gather info works.
Our reflections
It was useful to talk about changes, instead of writing. Experiences and ideas of all team members can be shared. Use the journal questions as basis for discussion.
The total info that the team knows intuitively seems to be sufficient to collect OM in a monthly meeting (it takes 4h / month)
The collected OM data seemed more robust when discussed and questioned by the entire team, rather than asking individuals to fill a form.
It is unclear if the depth of data is sufficient for robust OM analysis.
A recording device is useful to check the info when writing the report
In Feb 2011 we didn't discuss how to report on OM (language used for notes and reports)
It'd be good to verify information on behaviour changes with visits in the field.
Adjustments and follow-up
Small adjustment in OM framework: some progress markers have been reformulated/removed;
We merged the 4 outcome journals into one facilitator guide;
We added questions on capacity building and gender;
We agreed to have monthly OM meetings;
Our meetings are in Portuguese (but our notes are in English (by Mila and Birgit) - notes are translated into Portuguese to share with team members before next meeting.
Qualitative data file: One document is monthly updated with the notes of each meeting and info from IP reports.
Field visits: Based on most recent OM meetings, to follow up and verify data.
The final version of the report gives a short description including data source and data period (see example in the presentation).
Q: You are collecting info on monthly meetings - just on info shared by extension officers (-> no, the whole CARE team) - how accurate is it then? A: This is a very good question. We are talking about changes we perceive but we need the M&E officer to go to the field and verify that data. Q: In India. data collection is a good way to interact with communities to exchange ideas, is it also happening there? A: ??
When we have the team meetings we discuss what's going well or not and change our activities based on our OM work; Q: Your OM work is a lot about anecdotes - to what extent can you make any analysis and conclusions on the basis of this? Do you note down the questions that this raises etc.? A: It is not structural. When we have a question we discuss it outside OM meetings. As a team we sometimes run into other issues and we partly address those but not in a structural way. Q: If you would this every 3 months, would that affect your process? Would you lose clarity or would it highlight changes more clearly? A: When collecting data, we felt it was robust but it might too robust and detailed and we could consider do this more or less frequently... Q: How many months have you done this for? A: Since February 2011 and we reflected back until one year before. Since Feb. 2012 we have them on a monthly basis so we have had them 5 times.
Jharkhand
Following the same reporting system as in Rajasthan.
Our information flow starts from goat keepers using a record card for each buck. This buck card is the starting point, recording level of services.
The next level of information is the field guide, including 8 buck keepers. This compiles the record of each areaL number of vaccinations, castrations etc. and village meetings. Each FG conducts a village meeting every month looking at how many villagers attend.
Next level: field supervisor - he collects info and compiles it and gives feedback at monthly meetings. There are 4 field supervisors who report info to share feedback with ??
We don't have a full IP. Aside from IP, we have the same set up as in Rajasthan
Q: How useful have you found the information - do you have examples of how you used it (e.g. feedback from field supervisors)? A: Castration is going well in clusters 1 & 2 but not in 3 & 4. When you get in the details of that, you find out that farmers are selling goats more in 1& 2. Now we will bring farmers from 3 & 4 to 1 & 2. We have also registered less mortality through PPR vaccination brought up from last year. Q: Is the data from JAR analysed in the same way as in RAJ? A: Not yet. We don't have field operators in the field in JAR. Q: There are interactions with traders - are you capturing that information with them e.g. a short note? e.g. 1 Boundary partner could be about the 'enabling environment' (e.g. an NGO working on goats in the same province) and perhaps you make a note about this... A: It's not being done by our field staff right now. We could document that information and interactions with traders.
Does that raise any question about anything innovative, surprising, holding potential for your own work?
Impressions about the session (Roberto)
Most information is related to producers.
Qualitative information collected but not yet clear how to capture and analyse it.
India information collected by field guides and in Mozambique by extension officers - who collects that information?
Other impressions:
Interesting differences: teams have their own strategy for how to collect data. e.g. in Moz they are changing what extension workers know by heart. They triangulate by field meetings and facilitation. In India they use BAIF data and extract info for progress markers. Validity problem: they depend on field guides - but what is their capacity?
We have to collect all data that is important to monitor the progress - the diversity of methods is ok but we have to focus on important data for progress markers;
We have to think about how to analyse this data - have clear data about how and who, how frequently we analyse this data...
How much we are involving boundary partners in this analysis. How do they perceive that change? Perhaps in IP meetings we can present our findings and ask boundary partners to react on this. Every 6 months we could have a part of the IP meeting looking at what's going well or not and what type of change is happening.
Producers etc. are the continuous partners. Traders are clear: they will step in later, when they see concrete opportunities. But we need to maintain linkages. We have to pay attention to the timeline in engaging these different actors.
An IP is set up to strengthen linkages between actors but right now we are focusing on producers --> it depends on the frequency. Until now we have nothing special / different to offer. Households are involved from the start but traders not.
We are thinking about how to analyse and use the analysis of the data - something to think about later.
Wrt to involvement of field guides in info collection, validity etc. there is a close supervision of their work. The role of FGs is much bigger than the project itself. They have to be model farmers and a source of information. The FGs are not paid staff, they are part of the community and are expected to be part of it. In the process, their capacity is being developed. FGs will have opportunities to get back to other farmers as model farmers (mentoring approach). A critical issue about goat farmers is that they are shy etc. but these FGs could help people clarify their doubts etc. It's potentially a very powerful link between community members.
Close of the session
Very good preparation work by all;
Tezira's view on comms/outputs:
Great that you are documenting your work;
Great that we are collecting some information to inform the OM community globally;
Reporting of success stories is critical because it's one of the outcomes;
Later we'll talk about research communication which is important;
Comms is not just products, it's also about the process of engagement and I note that you have regular IP meetings, discussions with paravets etc. and I would like to command you on this.
Dr. Hedge's close:
Thank you Ann and Ewen for this unique way of introducing ourselves and practical approach;
In little time we were able to develop ideas for documentation on the process, technology, communication and extension;
We helped identify the 4 important OM questions and despite warmth and lack of power you were able to contribute effectively.
We were very happy that the exercise done 15 months ago was taken seriously and we have identified areas to change without damaging fundamentals.
We have taken this process further - I'd like to thank our 3 teams to manage the teams. My compliments to you.
imGoats project: Learning and reflection workshop
Udaipur, India, 2-6 July 2012
How did we apply Outcome Mapping?
Table of Contents
Rajasthan
Not every progress marker needs to be developed for every boundary partner.1. Role and importance of OM or what did you do instead and why?
2. What modifications brought to the OM methodology?
We made changes to group meeting registers for easy compilation. Right now it is a long narrative done in local language. Now we look at key issues, what issues to take up in IP meetings, what issues to bring from IP to group meetings, formatted in bullet points (easier to compile later).
3. Plan to gather monitoring data?
E.g. we're looking at whether our group is meeting without any facilitator. It's still too early to say if this works but now, after running tests at village level, we find out that some issues were addressed in villages (or not) so we get an idea of whether a group is active and mature or not.
Now we also get the amount of goats per sale but we can only say once a year if someone is selling more goats (otherwise it would be too time consuming).
The timeline is an important issue to keep in consideration.
Jharkhand has a similar approach.
Q: Are there any plans to move beyond production to trace changes among other boundary partners?
A: We haven't reached the state where traders come and purchase. We will have traders come and buy at the end of this year only. We are collecting market information etc. and we will provide this info to traders later. We will capture other information later.
Q: To what extent do existing forms capture other progress markers? Does this allow you to look at a set of progress markers?
A: If we have achieved 50%, we think it's a high level change. The information is going to group members but not many issues are flagged for consideration in IPs. That part is weak. (who is checking if progress markers are achieved? A: I (Ramkumar) am entering the data and provide guidance to code and compile this into the framework).
Q: Monthly analysis is a lot of work - we should discuss this... Is it really useful? Is it worth it?
A: This info is already collected by BAIF and we continue to use it. We have good discussions with FGs now because we can point out to issues we see from that data. This goes beyond anecdotal information. Field guides are key.
Q: Can you say more about the progress markers identified?
A: When people follow action plans etc. it's another progress marker. There is a weakness we identified about this marker. Information flows from IP to the groups happen but not the other way around, so we assess ourselves poorly about that.
Q: Are there groups that have reached the state of 'adult groups'?
A: That will happen. Group maturity is taking longer than anticipated.
Q: You are collecting a lot of quantitative data but shouldn't you also measure the internal management capacity of the group?
A: We have some action plans prepared in IP meetings. You have some information about why some people participated to meetings or not etc. We are now coding some of these statements.
Q: Who is collecting the data?
A: Field guides collect all data
Q: You gave them training?
A: Yes
IP reports: field staff prepare this in Hindi and send it to Ramkumar for
Mozambique
We made an overview of data that CARE collects which comprises 5 information types:
- Producer group registration (once, group level),
- Participant registration (once, individual),
- Paravet participation (twice, for training),
- Register of producer trainings (group, at every training),
- Register of goats sales at fairs (every fair),
- Participatoin of students in productivity tool (by paravets, who are starting to collect data)
IP reports are collected at every meeting (written by Birgit).Additional data:
Excel sheet of the progress markers (includes the scoring exercise of yesterday)
What we did: Before using the journal we had a team meeting with the objective 'to discover what OM information the team can come up with and test if the idea of a monthly meeting to gather info works.
Our reflections
It was useful to talk about changes, instead of writing. Experiences and ideas of all team members can be shared. Use the journal questions as basis for discussion.
The total info that the team knows intuitively seems to be sufficient to collect OM in a monthly meeting (it takes 4h / month)
The collected OM data seemed more robust when discussed and questioned by the entire team, rather than asking individuals to fill a form.
It is unclear if the depth of data is sufficient for robust OM analysis.
A recording device is useful to check the info when writing the report
In Feb 2011 we didn't discuss how to report on OM (language used for notes and reports)
It'd be good to verify information on behaviour changes with visits in the field.
Adjustments and follow-up
- Small adjustment in OM framework: some progress markers have been reformulated/removed;
- We merged the 4 outcome journals into one facilitator guide;
- We added questions on capacity building and gender;
- We agreed to have monthly OM meetings;
- Our meetings are in Portuguese (but our notes are in English (by Mila and Birgit) - notes are translated into Portuguese to share with team members before next meeting.
- Qualitative data file: One document is monthly updated with the notes of each meeting and info from IP reports.
- Field visits: Based on most recent OM meetings, to follow up and verify data.
The final version of the report gives a short description including data source and data period (see example in the presentation).Q: You are collecting info on monthly meetings - just on info shared by extension officers (-> no, the whole CARE team) - how accurate is it then?
A: This is a very good question. We are talking about changes we perceive but we need the M&E officer to go to the field and verify that data.
Q: In India. data collection is a good way to interact with communities to exchange ideas, is it also happening there?
A: ??
When we have the team meetings we discuss what's going well or not and change our activities based on our OM work;
Q: Your OM work is a lot about anecdotes - to what extent can you make any analysis and conclusions on the basis of this? Do you note down the questions that this raises etc.?
A: It is not structural. When we have a question we discuss it outside OM meetings. As a team we sometimes run into other issues and we partly address those but not in a structural way.
Q: If you would this every 3 months, would that affect your process? Would you lose clarity or would it highlight changes more clearly?
A: When collecting data, we felt it was robust but it might too robust and detailed and we could consider do this more or less frequently...
Q: How many months have you done this for?
A: Since February 2011 and we reflected back until one year before. Since Feb. 2012 we have them on a monthly basis so we have had them 5 times.
Jharkhand
Following the same reporting system as in Rajasthan.- Our information flow starts from goat keepers using a record card for each buck. This buck card is the starting point, recording level of services.
- The next level of information is the field guide, including 8 buck keepers. This compiles the record of each areaL number of vaccinations, castrations etc. and village meetings. Each FG conducts a village meeting every month looking at how many villagers attend.
- Next level: field supervisor - he collects info and compiles it and gives feedback at monthly meetings. There are 4 field supervisors who report info to share feedback with ??
We don't have a full IP. Aside from IP, we have the same set up as in RajasthanQ: How useful have you found the information - do you have examples of how you used it (e.g. feedback from field supervisors)?
A: Castration is going well in clusters 1 & 2 but not in 3 & 4. When you get in the details of that, you find out that farmers are selling goats more in 1& 2. Now we will bring farmers from 3 & 4 to 1 & 2. We have also registered less mortality through PPR vaccination brought up from last year.
Q: Is the data from JAR analysed in the same way as in RAJ?
A: Not yet. We don't have field operators in the field in JAR.
Q: There are interactions with traders - are you capturing that information with them e.g. a short note? e.g. 1 Boundary partner could be about the 'enabling environment' (e.g. an NGO working on goats in the same province) and perhaps you make a note about this...
A: It's not being done by our field staff right now. We could document that information and interactions with traders.
Does that raise any question about anything innovative, surprising, holding potential for your own work?
Impressions about the session (Roberto)
- Most information is related to producers.
- Qualitative information collected but not yet clear how to capture and analyse it.
- India information collected by field guides and in Mozambique by extension officers - who collects that information?
Other impressions:Close of the session